Lexicon Development for Varieties of Spoken Colloquial Arabic

In Arabic speech communities, there is a diglossic gap between written/formal Modern Standard Arabic (MSA) and spoken/casual colloquial dialectal Arabic (DA): the common spoken language has no standard representation in written form, while the language observed in texts has limited occurrence in speech. Hence the task of developing language resources to describe and model DA speech involves extra work to establish conventions for orthography and grammatical analysis. We describe work being done at the LDC to develop lexicons for DA, comprising pronunciation, morphology and part-of-speech labeling for word forms in recorded speech. Components of the approach are: (a) a two-layer transcription, providing a consonant-skeleton form and a pronunciation form; (b) manual annotation of morphology, part-of-speech and English gloss, followed by development of automatic word parsers modeled on the Buckwalter Morphological Analyzer for MSA; (c) customized user interfaces and supporting tools for all stages of annotation; and (d) a relational database for storing, emending and publishing the transcription corpus as well as the lexicon. 1. The status of Dialectal Arabic Since the mid-20 century, the term diglossia has been used to describe the sociolinguistic situation in Arabic speech communities (Ferguson, 1959): in each Arabicspeaking region, there are two distinct language systems in use. Modern Standard Arabic (MSA) is the sole system for official communication in government, news reporting and academia. While MSA is used orally in speeches, broadcast news and formal settings, only a small minority of the population has practical experience or facility in speaking it; for most users of MSA, it is like a second language, somewhat related to their primary spoken language, and its use consists mainly if not exclusively in reading, writing and listening. Spoken Arabic dialects are the primary languages in these communities, but their usage is almost exclusively oral. All formal instruction in reading and writing is conducted in and for MSA: being literate means reading and writing MSA. Differences between MSA and DA involve a variety of diachronic sound changes affecting both manner and place of articulation for several consonants, as well as alterations in derivational and inflectional morphology that may reflect a restructuring of some underlying paradigms. Efforts to establish an explicit standardization for DA orthography and grammar have arisen only recently and are very rare; for the most part, such standardization does not exist. So, even though DA speakers may be familiar with a writing system and a wide range of textual resources, this is only marginally related to their daily usage of speech. In creating corpora and linguistic annotations for DA – to address the spoken language for purposes of human language technologies – we lack some of the basic underlying resources that are typically available in other literate languages. We must first establish an orthography that will serve as an adequate “canonical” written form, but still preserve evidence of significant pronunciation variants where possible, because in the absence of an established orthographic standard, and with only limited samples of speech to work from, observed variations could have equal standing as the basis for canonical spellings and further analysis of the language. Next, given the complex morphological structure of DA, we need to develop a lexicon that will support the automation of morphological analysis, by creating a sufficient body of manual annotations. In the process, we need a means to assure that all annotations can be revisited, amended and refined in an efficient and reliable manner while both transcription and manual analysis are in progress, with suitable feedback to annotators as further transcription and manual analysis are done. 1.1. Issues for MSA-based annotation of DA The differences between MSA and DA created by diachronic sound changes are significant enough that MSA is unsuitable as a standard orthography for DA. Still, the Arabic script-based writing system is familiar to all literate speakers of DA, and there is a fairly large base of common cognate vocabulary between MSA and DA, making the use of Arabic script, and of orthographic practices that closely resemble those of MSA, an effective means for transcription of DA speech (Maamouri et al., 2004a,b). The keyboard layout for these characters can be learned fairly quickly, and transcribers find it easier to read and verify their typing when it is presented in Arabic script, rather than Latin/ASCII transliteration. It is also useful to distinguish two forms for each word: a “consonant skeleton” form, consistent with standard orthographic practice in MSA, and a “diacritized” form, using the common Arabic diacritic marks (for short vowels, consonant gemination, etc), to represent pronunciation. For morphological analysis, the situation is more difficult. Given that DA is directly related to MSA, and we have very good tools for analyzing the morphology of MSA, we first made an attempt to adapt the Buckwalter Morphological Analyzer (Buckwalter, 2004) so that it could provide candidate analyses of DA word forms. The task for annotators, we hoped, would then be simply identifying which of several possible analyses was the appropriate one for a given word. When we tried this approach on a set of transcripts drawn from a corpus of Levantine Arabic telephone conversations (Maamouri et al., 2006), we found that the differences between MSA